Google study says GPT-5 and Gemini 3 often know facts but fail to recall them

Google Research says a large share of factual errors in frontier language models may come from retrieval failure rather than missing knowledge. In a study titled "Empty Shelves or Lost Keys?" and accepted at ICML 2026, researchers found that GPT-5, Gemini 3 and other tested models had already stored 95%–98% of benchmark facts in their parameters, yet still failed to produce 26%–34% of those facts in direct question answering. Even with thinking enabled, 11%–12% remained inaccessible. The work introduces a "knowledge states" framework and a new benchmark called WikiProfile, built from 2,150 English Wikipedia facts and 21,500 associated questions. Google evaluated 13 models across the Gemini 3, GPT-5, GPT-4.1 and Gemma 3 families, with and without thinking, sampling each question eight times for roughly 4.5 million responses. The results point to two major choke points: rare facts and reversed questions. The paper argues that in both cases the issue is often not that the model never learned the fact, but that it struggles to retrieve it when wording or direction changes. Google also reports that thinking helps recover 40%–65% of facts that were stored but initially unreachable, while helping only 5%–15% on facts that were never stored, suggesting thinking can function as a recall aid rather than only a reasoning tool.